…failure

The open5gs_vm image (id: 260) advertises kex-strict-s-v00@openssh.com but
fails to complete the handshake when strict KEX mode is activated, causing
Stage 3 (wait_for_connection) to time out with "Connection closed by UNKNOWN
port 65535". Adding -o KexAlgorithms=curve25519-sha256 prevents the client
from engaging strict mode, which allows the key exchange to complete.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Follows the installation fix in commit 6a7128e; the README and result
template still referenced the old pinned version.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… in public.yaml

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

unattended-upgrades holds /var/lib/dpkg/lock-frontend on VM boot,
causing apt installs to fail with rc 100 when Ansible runs immediately
after the VM becomes reachable.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…meout

Passive wait could timeout (>5 min) if unattended-upgrades was running.
Stop the service first so the lock is released immediately.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Stopping unattended-upgrades via systemd sends SIGTERM but the in-flight
dpkg operation keeps holding lock-frontend until it completes. Use fuser -k
to immediately kill any process holding the dpkg/apt locks, then repair
any interrupted dpkg state before proceeding with apt installs.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… user profile

All VM Terraform templates used \$USER[SSH_PUBLIC_KEY], which expands to the
SSH key stored in the OpenNebula account of the deploying user. When that user
is a human operator rather than Jenkins, the provisioned VM gets the wrong key
in authorized_keys and Jenkins can never authenticate.

Read the key at pipeline time from /var/lib/jenkins/.ssh/id_ed25519.pub via an
Ansible lookup (always runs on the controller) and inject it as
jenkins_ssh_public_key. Replace \$USER[SSH_PUBLIC_KEY] with this variable in
all 23 VM component templates.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ents

ProxyJump (-J) does not forward -o KexAlgorithms to the bastion hop; it uses
system defaults for the proxy connection. Bastions running the Open5GS appliance
image advertise kex-strict-s-v00@openssh.com, which causes the proxy hop to fail
with "kex_exchange_identification: Connection closed by remote host" even when
KexAlgorithms=curve25519-sha256 is set for the target connection.

Replace -J with an explicit ProxyCommand that runs ssh+nc and carries
KexAlgorithms=curve25519-sha256 for the bastion hop, and also sets it for the
final hop. Fixes SSH connectivity for all 21 VM components that use a bastion
proxy. Multi-hop variants (oneKE, open5gs_k8s, ocf) are left unchanged.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…nection timeout

Ubuntu VMs deployed via OpenNebula one-appliance trigger unattended-upgrades at
first boot, which runs in the background for ~16 minutes and restarts openssh-server
mid-session. This causes Ansible wait_for_connection to fail with
"Connection to <ip> closed by remote host" before provisioning can start.

Two-part fix applied to all VM components:
1. Add START_SCRIPT_BASE64 to the OpenNebula VM context: stops and masks
   unattended-upgrades, apt-daily-upgrade, and apt-daily services at first boot,
   before the apt-daily-upgrade.service timer fires at ~30 seconds post-boot.
2. Increase wait_for_connection timeout from 200s to 600s (10 min per phase,
   ~20 min total) as a safety net for any remaining boot-time delays.

For ks8500_runner and loadcore_agent the existing START_SCRIPT is extended with
the unattended-upgrades stop/mask commands prepended.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… open5gs_vm

The previous START_SCRIPT_BASE64 masked apt service units but not their
timers (apt-daily.timer and apt-daily-upgrade.timer), allowing
unattended-upgrades to run at first boot via the timer path and restart
sshd during Ansible provisioning.

Also enables ansible_pipelining on open5gs_vm to avoid the strict-KEX
bug (CVE-2023-48795 mitigation in OpenSSH 8.9) which causes SFTP to
hang when the default KEX proposal includes kex-strict-c-v00@openssh.com.
With pipelining on, Ansible sends module code via SSH exec stdin instead
of SFTP, bypassing the hang entirely.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… transfers

With pipelining enabled, the template module serialises the full hostvars
dict and sends it through the nc ProxyCommand chain. This is large enough
to stall the pipe. The shell module was unaffected because its payload is
tiny. Removing pipelining restores SFTP-based file transfer, which works
correctly with the existing KexAlgorithms=curve25519-sha256 fix.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…-proxied connections

The Jenkins global ~/.ansible.cfg sets ControlMaster=auto with ControlPersist=60s.
When Ansible connects to the open5gs_vm via nc ProxyCommand, the ControlMaster is
created but the nc tunnel drops after the idle period between wait_for_connection
and the first real task. This leaves a stale socket file that causes all subsequent
SSH connections to hang indefinitely.

-S none in ansible_ssh_common_args uses the OpenSSH -S flag which directly
overwrites options.control_path, bypassing the first-wins rule that applies to -o
options. This definitively disables connection sharing regardless of ansible.cfg.

ansible_pipelining: false explicitly overrides the global transfer_method = piped
setting in ansible.cfg so SFTP is used for file transfers instead of piped stdin.

Also: clean up any stale socket at the start of Stage 3 as a belt-and-suspenders
measure, and increase wait_for_connection timeout to 1800s to handle slow VM boots
where cloud-init apt operations can take over 16 minutes.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…p task

The delegate_to: localhost task in Stage 3 was inheriting ansible_user=jenkins
from the open5gs_vm inventory entry, causing Ansible to SSH to localhost rather
than running the file deletion locally. Added vars: ansible_connection: local to
force local execution.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ommand

ansible_pipelining: false forced sftp-based file upload for become: true tasks,
which opens a separate SSH sub-connection through the nc ProxyCommand. That
connection is closed by the remote host. The uma-test site succeeded with
pipelining enabled — the stale ControlMaster was the real root cause, now fixed
with -S none. Pipelining through nc ProxyCommand works for all task sizes here.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…etplan resilience

ControlMaster=no caused every task to open a fresh SSH tunnel. When netplan apply
briefly disrupts the VXLAN interface (192.168.199.x), the next connection attempt
catches the window and fails with "Connection closed by remote host." Using
ControlMaster=auto with ControlPersist=3600 keeps a single tunnel alive through
the disruption. The original stale-socket bug (60s expiry) is prevented by the
longer persist window plus the cleanup task that removes old sockets at Stage 3
start. Socket path is explicit (-S /tmp/cm-<ip>-22-jenkins) so cleanup is exact.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…n apply

ControlMaster=auto caused silent socket failures (empty error, earlier than
previous failures). Reverting to -S none/-o ControlMaster=no which was stable
through task 16 in the previous run.

The remaining blocker was netplan apply briefly disrupting the VXLAN interface
(192.168.199.x), causing the next apt_repository task to get "Connection closed
by remote host". Adding wait_for_connection after netplan apply (same when
condition) ensures the SSH connection is re-established before apt tasks run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

On slow-booting VMs (>15 min), unattended-upgrades runs during the cloud-init
window and may restart sshd after wait_for_connection succeeds. The five
variable-loading tasks that follow run on the Ansible controller (no SSH), so
they appear OK. The first real remote task (Set site ssh key) then hits the
restarting sshd and gets "Connection closed by remote host."

A second wait_for_connection (delay 5s, timeout 300s) after variable loading
but before the first remote task absorbs any sshd restart window.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

On slow-booting VMs unattended-upgrades runs during cloud-init and periodically
restarts sshd, which kills active Ansible connections. Adding a shell task right
after the initial wait_for_connection to stop unattended-upgrades and release
dpkg locks. ignore_unreachable/ignore_errors tolerates the case where sshd dies
mid-task; the follow-up wait_for_connection absorbs the restart window before
any real configuration work begins.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…sshd disruption

Replace systemctl stop with mask --now for unattended-upgrades and
apt-daily timers in both the pre-install stage and the installation
block. Also write a needrestart override (restart=l) so that needrestart
does not auto-restart sshd after mongodb-org or open5gs packages are
installed, which was causing UNREACHABLE failures mid-installation.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

… tasks

cloud-init restarts sshd multiple times during early VM boot. The
previous wait_for_connection only confirms reachability at that instant;
a second restart window during cloud-init can hit the Create-new-user
task. Adding cloud-init status --wait (with ignore_unreachable and a
recovery wait_for_connection) ensures cloud-init is fully done before
any state-changing remote tasks run.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…nsible wait_for_connection

The ProxyCommand tunnels (Jenkins → bastion → VM via nc) were being killed
by a ~946-second TCP idle timeout on the client network because no data
flowed while Ansible waited for the ping module response. Each attempt
consumed the full 946 s before failing, causing the 1800-second
wait_for_connection timeout to be exceeded after only two retries.

Adds -o ServerAliveInterval=30 -o ServerAliveCountMax=10 to both the
outer ansible_ssh_common_args and the inner ProxyCommand ssh invocation
across all 21 VM component playbooks, keeping both the Jenkins→bastion
and the end-to-end SSH connections alive during idle periods.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

dpkg --configure -a runs the openssh-server post-install script which
calls invoke-rc.d ssh restart, dropping the Ansible SSH session and
causing UNREACHABLE. The task had no ignore_unreachable guard, so the
play failed.

Add a policy-rc.d stub that returns 101 (action refused) before running
dpkg --configure -a so invoke-rc.d refuses all service restarts during
package configuration. Remove it immediately after. Also add
ignore_unreachable: true as a safety net and an unconditional
wait_for_connection after the task to reconnect if sshd does restart.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…shd restart

dpkg --configure -a is called with policy-rc.d (exit 101) during the
early apt-masking step, before any non-ignored Ansible tasks. Any
pending openssh-server post-install script is resolved here without
restarting sshd, eliminating the deferred restart that caused
UNREACHABLE on Set TN ssh key as authorized in new user (Bug 18).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…figure

wait_for_connection timed out at elapsed=322s against timeout=120s on
raul13 (Bug 20): recovery after the guarded dpkg --configure -a
UNREACHABLE genuinely takes longer than 120s. Extend to 600s and
guard with ignore_unreachable/ignore_errors, then confirm stability
with a second bounded wait, matching the pattern already applied to
the pre-install stabilize checks in component_playbook.yaml (Bug 19).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…oy apply

The OpenNebula terraform provider hard-errors with "AR not found" when an
opennebula_virtual_network_address_range's AR was already destroyed as a
side effect of the first (ignored) apply, but the resource is still tracked
in state. Parse that failure for the affected VNet IDs and remove just the
matching stale AR entries from state before the second, must-succeed apply.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…RNAL error

The first observed failure mode ("AR (ID:Y) not found") is not the only one
OpenNebula's provider produces for an already-gone AR; a second run hit
"AR (ID:Y): OpenNebula error [INTERNAL]: [one.vn.free_ar] Address Range does
not exist" instead, which the previous regex didn't match. Match on the
common "Virtual network (ID:X): AR (ID:Y)" prefix instead of the trailing
error text, since that phrase is specific to this failure regardless of
wording.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…urces

The previous selectattr('values.virtual_network_id', 'in', stale_ar_vnet_ids)
pipeline silently matched nothing whenever virtual_network_id was stored as
an int in state (stale_ar_vnet_ids from regex_findall is always strings),
so the cleanup step never actually ran despite correctly identifying the
affected VNet IDs. Replace the selectattr/map pipeline with a direct loop
over state resources with an explicit `| string` cast in the per-item when
condition, which can't silently type-mismatch the way selectattr('in') did.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>

…sion

`.values` and (per-resource) `.values` again are dict keys in terraform
show -json's output, but Jinja2 dot-notation resolves them to Python's
built-in dict.values() method instead of the "values" key, since attribute
lookup is tried before key lookup. Combined with `| default([])`, this
silently produced an empty list rather than erroring, so the loop always
had zero items and "skipping" was the *expected* (but wrong) outcome for
every run so far. Switch both accesses to bracket notation, which resolves
against dict keys directly. Verified in isolation with a hand-crafted
terraform show -json payload before landing this.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>